Efficient data transfer

ABSTRACT

A determination is made to co-locate geographical dispersed data of a Relational Database Management System (RDBMS). Costs associated with the direction of moving a portion of the dispersed data from one site to another site are computed. A comparison of the costs is made to determine which direction to move the data to co-locate it at one site for optimal performance of a RDBMS operation.

BACKGROUND

Frequently, enterprises are maintaining portions of their informationassets at multiple geographically dispersed locations. For example, onefacility may need to capture its data locally because that data is usedthe most frequency by that facility. Sometimes the data relevant to allthe facilities of an enterprise are centrally located at a remote,external, and geographical facility. Each facility may have its own dataneeds, some of which may never be moved to the centrally locatedfacility. Moreover, in some cases, it is more efficient for the data tobe distributed across facilities but act together as one data warehouseas a single Relational Database Management System (RDBMS). That is,there are a variety of reasons why enterprise will have data assetsgeographically distributed and still be capable of issuing queries thataccess the entire set of data assets from multiple facilities.

However, transferring large amounts of data over network connections isa performance sink. The data may need transferred for a variety ofreasons, such as to perform a join operation on two database tableswhile processing a query where each table is located on differentservers over a network that are geographically dispersed. Moreover, eachserver may be missing one of the tables needed for the join operation.In this example and for the query to process properly, one of theservers needs to transfer its table relevant to a portion of the joinoperation to the other server having the remaining table relevant to theother portion of the join operation so that the join operation canefficiently process.

Presently, there is no sufficiently intelligent mechanism to selectwhich way the table being transferred over the network should move.Typically, there is likely just a default rule that an enterprisedeploys that moves one table in a predefined direction over the network.However, depending upon the type and quantity of data for the tablebeing transferred, this default rule can be a mistake by unnecessarilytaxing network bandwidth, resources, and significantly delaying resultsreturned form a query to an end user.

Some existing solutions attempt to alleviate the situation withelaborate caching techniques (which have to be continually updated andflushed to stay up-to-date with the ever changing data).

Therefore, there is a need to more efficiently transfer data in ageographically dispersed database environment when processing queriesthat rely on geographically dispersed data tables.

SUMMARY

In various embodiments, methods and a system for efficient data transferare presented. According to an embodiment, a method for efficient datatransfer is provided.

Specifically, a determination is made that a database operation needsaccess to data on a first server and a second server; the first serverand the second server are geographically dispersed over a networkconnection. Costs associated with moving a portion of the data from oneof the servers to a remaining one of the servers in performance of apart of the database operation are calculated. Finally, the portion istransferred from the server having a least amount of cost to the serverhaving a greater amount of cost for processing the part of the databaseoperation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an example configuration and situation forefficient data transfer, according to an example embodiment.

FIG. 2 is a diagram of a method for efficient data transfer, accordingto an example embodiment.

FIG. 3 is a diagram of another method for efficient data transfer,according to an example embodiment.

FIG. 4 is a diagram of a data transfer system, according to an exampleembodiment.

DETAILED DESCRIPTION

FIG. 1 is a diagram of an example configuration and situation forefficient data transfer, according to an example embodiment. It is notedthat the components of the example configuration are shown in greatlysimplified form with just components necessary for understating theembodiments presented herein presented for illustration.

The decision maker as to which direction data is transferred over ageographically dispersed RDBMS is implemented as one or more softwaremodules as executable instructions that execute on one or moreprocessors of a network, where those processors are specificallyconfigured to execute the software modules. For the discussion of theFIG. 1, the software modules are referred to as a “smart transferagent.”

In an embodiment, the smart transfer agent is integrated into one ormore components of an interface associated with the RDBMS.

In an embodiment, the smart transfer agent is a component of the DataAnywhere® product, distributed by Teradata®, Inc.

In a RDBMS where data can reside in disparate locations (differentservers that are co-located or different servers located in differentparts of the world (geographically dispersed)), some operations of theRDBMS may require that the data be co-located in the performance of someaspects of a query being processed against the RDBMS.

The smart transfer agent provides a mechanism to identify where thevarious portions of data associated with a query being performed in theRDBMS reside, how a portion of the query can be split, which directionto move one portion of the data to have it co-located with the otherportion of the data, and reconfigure the split portion of the query tooperate on the now co-located data.

Again, a major bottleneck with large-scale geographically dispersedRDBMSs is the transferring of large amounts of data from one networklocation to another network location in performance of a query againstthe RDBMS. The smart transfer agent solves this problem in an efficientmanner to transfer data over the network in a direction that is optimalin performance of the query.

Initially, the smart transfer agent determines that a portion of a querybeing processed in a RDBMS requires data that is not co-located (forexample a join operation is best for the query that uses the twodatabase tables of the RDBMS). Next, the smart transfer agent obtains apre-calculated cost for two tables that the smart transfer agentdetermined were best joined to process a particular portion of the query(could be the whole query as well). The cost (transfer cost) roughlyequates to the number of rows within the table having a set of dataneeded by the query. The transfer cost along with the column data typeassociated with the join operation is then used to calculate the overallcost of transferring data in the table (first table) being evaluated andlocated in a first location over a network to a second location, whichis not co-located and is geographically dispersed over a network fromthe first table at the first location. The transfer cost and the overallcost are added together to obtain a first server's transfer cost for thefirst server having the first table at the first location. This sameprocedure is repeated for the second table at the second location toobtain the second server's transfer cost in moving the second table overthe network to the first location. The two servers' transfer costs arethen compared by the smart transfer agent to drive the actual directionthat the data from the two tables will move.

In an embodiment, the servers' transfer costs can also be augmented byfactors that form a ratio. This is so because one of the servers mayinclude most data for the RDMS (central data location, referred hereinto as “home server”), all table definitions for the RDMS, and theapplication components that process against the RDMS. So, in order toeliminate unnecessary data transfers over the network, the aboveapproach is adjusted to more favorably weight a decision to move data inthe direction of the home server; rather than from the home server tothe external location. This will ensure that data is only transferredaway from the home server when the costs benefits of doing so aregreater than the cost of transferring smaller data to the larger data(at the remote site away from the home server), filtering that data (atthe remote site), and then transferring that data back to the homeserver for completion of query processing.

An example situation is now presented to illustrate the processing ofthe smart transfer agent with reference to the FIG. 1.

The highly simplified depiction of the RDMS in the FIG. 1 includes ahome server B having, initially, table B; and an external server A(geographically dispersed from the home server B and not co-located withthe home server B). External server A includes, initially, table A.

A query is being processed in the example situation where a query isbeing processed that needs to query data from both table A and table B.Thus, as discussed above, for this situation the smart transfer agentdetermines that because table A is geographically dispersed it is moreefficient to co-locate the data from tables A and B and then perform thequery. A mechanism to do this is to join tables A and B together into atemporary table with a result produced.

The smart transfer agent acquires the transfer costs and the overallcosts for each table A and B. Then, the smart transfer agent computeseach server's transfer cost. Additionally, each server's transfer costis then augmented by the ratio associated with costs of moving data fromthe home server B.

Suppose the tables are configured and have the following costs:

Server A (External Customer Data Warehouse)

-   -   Table A (1,000,000 rows of data) Transfer Cost of 100;        -   Column id (integer)        -   Column name (varchar 100)        -   Column description (varchar 500)

Server B (Home Server)

-   -   Table B (10,000 rows of data) Transfer Cost of 1;        -   Column id (integer)        -   Column location (varchar 100)        -   Column location Description (varchar 500)

If the query conditions defining all data from both table A and table Bwhere A.id=B.id smart transfer agent proceeds as follows based on theabove information for the tables:

1. Calculate the Cost of Table A: 100*(1+100+500)=60100;

2. Add up all Table costs for Server B: 601;3. Add up all Table costs for Server A: 60100

4. Calculate the Cost of Table B: 1*(1+100+500)=601; 5. Apply the HomeServer Ratio (2:3)

-   -   60100*2=120200; (home server B)    -   601*3=1803; (external server A)

The smart transfer agent then compares the results after applying thehome server ratio and determines how the query should be split andtransferred. In this case, the Data from table B (home server B) isselected and transferred to external server A and placed into atemporary table. Then, the query is rewritten where it selects fromtable A and joins with the new temporary table; the new temporary tablewas just created by the smart transfer agent. This query is executed andthe results will be transferred back to the home server B and stored ina return table appropriate for any segmentation process to continueoperating on the data, if so desired.

In an embodiment, the home server ratio is a configurable parameter tothe processing of the smart transfer agent.

In an embodiment, the home server ratio is obtained by the smarttransfer agent from a database table or known storage location.

In an embodiment, the home server ratio can be dynamically changed whilethe smart transfer agent executes, such as by updating the known storagelocation discussed in the latter embodiment (as one example).

FIG. 2 is a diagram of a method 200 for efficient data transfer,according to an example embodiment. The software module(s) thatimplement the method 200 are herein referred to as “smart transfermanager.” The executable instructions of the smart transfer manager areprogrammed in memory and/or non-transitory computer-readable storagemedium, which execute on one or more processors (specifically configuredto execute the smart transfer manager). Moreover, the smart transfermanager has access to one or more networks (wired, wireless, or acombination of both wired and wireless).

In an embodiment, the smart transfer manager is the smart transfer agentdiscussed above with reference to the FIG. 1.

In an embodiment, the smart transfer manager processes in parallel withother instances of the smart transfer manager over a massively parallelprocessing distributed RDBMS, each instance cooperating with the otherinstance (in some RDBMS transactions requiring the cooperation).

At 210, the smart transfer manager determines a database operation thatneeds access to data on a first server and a second server. The firstserver and the second server are geographically dispersed over a networkconnection from one another. That is, the first and second servers arenot co-located.

According to an embodiment, at 211, the smart transfer manageridentifies the database operation as a query having conditions forsearching both fa first database table of the first server and a seconddatabase table of the second server.

At 220, the smart transfer manager calculates costs for moving a portionof the data from one of the servers to a remaining server in performanceof part of the database operation.

In an embodiment of 211 and 220, at 221, the smart transfer managerrestates part of the database operation to access the table that wastransferred as the portion of the data to use a temporary tablerepresenting the table that was transferred from the server with theleast amount of cost.

According to an embodiment of 211 and 220, at 222, the smart transfermanager determines the cost as: a first transfer cost for the firsttable based on the number of rows in the first table and a secondtransfer cost for the second table based on a total number of rows inthe second table.

In an embodiment of 222 and at 223, the smart transfer managercalculates a first overall cost based on column attributes of the firsttable and calculates a second overall cost based on column attributes ofthe second table. In an embodiment, the augmentation is identifying thedata type for the first table column and the data type for the secondtable column. The data type provides an indication as to the size of thedata in the columns, for example an integer may be labeled as having adata size of 1; some column data types define their size (see theexample presented above with the FIG. 1). The sum of the sizes for thecolumn data types are added together, and then this sum is multiplied bythe number of rows in the table to obtain the transfer cost for a giventable. So, the first column data sizes are summed together to get atotal first column data size and the total first column data size ismultiplied the total number of rows in the first table to get the firstoverall cost. Similarly, the second column data sizes are summedtogether to get a total second column data size and the total secondcolumn data size is multiple by the total number of rows in the secondtable to get the second overall cost. This was discussed above orimplied above with the illustration of the FIG. 1 and is one mechanismunder the embodiment of 223 that can be used to calculate the over costof each table. The column attributes in this example are the data sizesof the data that populates the column.

In an embodiment of 223 and at 224, the smart transfer manager obtains aratio that weights any transfer between the servers based on attributesof each server.

In an embodiment of 224 and at 225, the smart transfer managermultiplies a first factor from the ratio and the first overall cost toobtain a first server transfer cost and multiplies a second factor fromthe ratio and the second overall cost to obtain a second server transfercost.

In an embodiment of 225 and at 226, the smart transfer manager comparesthe first server transfer cost to the second server transfer cost toidentify the server having the least amount of cost and to identify theserver having the greater amount of cost.

The processing discussed at 222-226 was discussed above with referenceto the FIG. 1 where an example scenario was presented for comprehensionof the calculations.

At 230, the smart transfer manager transfers the portion of the datafrom the server having the least amount of cost to the server having thegreater amount of cost where the part of the database operation isprocessed.

In an embodiment, at 231, the smart transfer manager sends the part ofthe database operation with the portion of the data to the server withthe greater amount of cost as a restated part of the database operationto process on the server having the greater amount of cost.

In an embodiment of 231 and at 232, the smart transfer manager revisesthe part of the database operation to create the restated part of thedatabase operation to ensure that the restated part of the databaseoperation operates on a temporary table created for housing the portionof the data transferred to the server with the greater amount of cost.That is, the reference handle for the database operation needs changedbecause the reference to the portion of data now resides within theenvironment of the server with the greater amount of cost and not whereit originally resided, which was on the server with the least amount ofcost.

According to an embodiment, at 240, the smart transfer manager receivesresults from the server with the greater amount of cost after the serverwith the greater amount of cost has processed the part of the databaseoperation that it was supposed to process.

In an embodiment of 240 and at 241, the smart transfer manager providesthe results to a remaining part of the database operation for thatoperation to complete its processing on the server having the leastamount of cost.

FIG. 3 is a diagram of another method 300 for efficient data transfer,according to an example embodiment. The software module(s) thatimplement the method 300 are herein referred to as “data transfermanager.” The executable instructions of the data transfer manager areprogrammed in memory and/or non-transitory computer-readable storagemedium, which execute on one or more processors (specifically configuredto execute the data transfer manager). Moreover, the data transfermanager has access to one or more networks (wired, wireless, or acombination of both wired and wireless).

The data transfer manager presents another, and perhaps, enhancedprocessing perspective record linker, presented above with respect tothe FIGS. 1 and 2.

In an embodiment, the data transfer manager is the smart transfer agentof the FIG. 1.

In an embodiment, the data transfer manager is the smart transfermanager of the FIG. 2.

In an embodiment, the data transfer manager processes in parallel withother instances of the data transfer manager over a massively parallelprocessing distributed RDBMS, each instance cooperating with the otherinstance (in some RDBMS transactions requiring the cooperation).

At 310, the data transfer manager determines to transfer a firstdatabase table located on a first server to a second server having asecond database table. The first and second servers are not co-locatedwith one another.

In an embodiment, at 311, the data transfer manager resolves to transferthe first table based on a cost analysis of transferring the first tablefrom the first server to the second server and transferring the secondtable from the second server to the first server.

At 320, the data transfer manager revises a query submitted forprocessing on the first server to execute a revised query on the secondserver.

In an embodiment, at 321, the data transfer manager produces the revisedquery based on the first table having a different reference handle whenaccessed on the second server.

At 330, the data transfer manager transfers the first table and therevised query to the second server for executing using the second table.The first server and the second server are geographically dispersed fromone another over a network connection.

According to an embodiment, at 331, the data transfer manager instructsthe second server to create a temporary table for housing data of thefirst table on the second server.

According to an embodiment, at 340, the data transfer manager executesas a feature of a distributed RDBMS. This was discussed above withreference to the FIG. 1.

FIG. 4 is a diagram of a data transfer system 400, according to anexample embodiment. Some components of the data transfer system 400 areimplemented as one or more software modules having executableinstructions that are programmed within memory and/or non-transitorycomputer-readable storage media and that execute on one or moreprocessing nodes (processors) of a network. Furthermore, the datatransfer system 400 has access to one or more networks. The network iswired, wireless, or a combination of both wired and wireless.

The data transfer system 400 implements, inter alia, the techniquespresented above in the FIGS. 1-3, which are incorporated by referenceherein with the discussion of the data transfer system 400.

The data transfer system 400 includes at least one processor 301 of aRDBMS and a smart transfer agent 302.

The processor(s) 301 have access to memory and/or non-transitorycomputer-readable storage media to execute the smart transfer agent 302.

The smart transfer agent 302 is configured to: execute on theprocessor(s) 301; perform a cost analysis for a direction to transfer adatabase table in performance of executing a query, the direction beingfrom a particular one of the servers and to a remaining one of theservers; and transfer the database table in the direction to a serverhaving a greater amount of the cost determined by the cost analysis forexecuting the query on that server.

According to an embodiment, the smart transfer agent 302 is configuredto receive results back from the server having the greater amount of thecost in response to executing the query on that server.

In an embodiment of the latter embodiment, the smart transfer agent 302is configured to provide the results to a database applicationassociated with the query on the server having a least amount of thecost.

The above description is illustrative, and not restrictive. Many otherembodiments will be apparent to those of skill in the art upon reviewingthe above description. The scope of embodiments should therefore bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

1. A method, comprising: determining, by a processor, a databaseoperation that needs access to data on a first server and a secondserver, the first server and the second server are geographicallydispersed over a network connection; calculating, by the processor,costs associated with moving a portion of the data from one of theservers to a remaining one of the servers in performance of a part ofthe database operation; and transferring, by the processor, the portionfrom the server having a least amount of cost to the server having agreater amount of cost where the part of the database operation isprocessed.
 2. The method of claim 1, wherein determining furtherincludes identifying the database operation as a query having conditionsfor searching both a first database table located on the first serverand a second table located on the second server.
 3. The method of claim2, wherein calculating includes restating the part of the databaseoperation to access the table that was transferred as the portion of thedata to use a temporary table representing the table that wastransferred to the server having the least amount of cost.
 4. The methodof claim 2, wherein calculating further includes determining the costsas: a first transfer cost for the first table based on a total number ofrows in the first table and a second transfer cost for the second tablebased on a total number of rows in the second table.
 5. The method ofclaim 4, wherein determining further includes calculating a firstoverall cost by augmenting the first table transfer cost based on columnattributes of the first table and calculating a second overall cost byaugmenting the second table transfer cost based on column attributes ofthe second table.
 6. The method of claim 5, wherein calculating furtherincludes obtaining a ratio associated with weighting any transferbetween the servers based on attributes of each server.
 7. The method ofclaim 6, wherein obtaining further includes multiplying a first factorfrom the ratio and the first overall cost to obtain a first servertransfer cost and multiplying a second factor from the ratio and thesecond overall cost to obtain a second server transfer cost.
 8. Themethod of claim 7, wherein multiplying further includes comparing thefirst server transfer cost to the second server transfer cost toidentity the server having the least amount of cost and to identity theserver having the greater amount of cost.
 9. The method of claim 1,wherein transferring further includes sending the part of the databaseoperation with the portion of data to the server with the greater amountof cost as a restated part of the database operation to processing onthe server with the greater amount of cost.
 10. The method of claim 9,wherein sending further includes revising the part of the databaseoperation to create the restated part to ensure that the restated partoperates on a temporary table created for housing the portion of thedata transferred to the server with the greater amount of cost.
 11. Themethod of claim 1 further comprising, receiving results from the serverwith the greater amount of cost after the server with the greater amountof cost has processed the part of the database operation.
 12. The methodof claim 11, wherein receiving further includes providing the results toa remaining part of the database operation to complete processing of thedatabase operation.
 13. A method, comprising: determining, by aprocessor, to transfer a first database table located on a first serverto a second server having a second database table; revising, by theprocessor, a query submitted for processing on the first server toexecute as a revised query on the second server; and transferring, bythe processor, the first database table and the revised query to thesecond server for execution using the second database table, wherein thefirst server and the second server are geographically dispersed from oneanother over a network connection.
 14. The method of claim 13,determining further includes resolving to transfer the first databasetable based on a cost analysis of transferring the first database tablefrom the first server to the second server and transferring the seconddatabase table from the second server to the first server.
 15. Themethod of claim 13, wherein revising further includes producing therevised query based on the first database table having a differentreference handle when accessed on the second server.
 16. The method ofclaim 13, wherein transferring further includes instructing the secondserver to create a temporary table for housing data of the firstdatabase table on the second server.
 17. The method of claim 13 furthercomprising, processing the method as a feature of a distributedRelational Database Management System.
 18. A system, comprising:processors arranged to process a distributed Relational DatabaseManagement system over a first server and a second server; and a smarttransfer agent configured to: (i) execute on one or more of theprocessors; (ii) perform a cost analysis for a direction to transfer adatabase table in performance of executing a query, the direction beingfrom a particular one of the servers and to a remaining one of theservers; and (iii) transfer the database table in the direction to aserver having a greater amount of the cost determined by the costanalysis for executing the query on that server.
 19. The system of claim18, wherein the smart transfer agent is further configured to (iv)receives results back from the server having the greater amount of thecost in response to executing the query on that server.
 20. The systemof claim 19, wherein the smart transfer agent is further configured to(v) provides the results to a database application associated with thequery on the server having a least amount of the cost.